The present invention relates to the field of data processing, and more particularly to a method, computer program product and system for data processing in a multi-threaded processing arrangement.
Today, data processing arrangements (software and/or hardware) typically have an inherent requirement to process large volumes of data from a variety of different sources. By way of example, a file or database may contain many TeraBytes (TB) of data, not all of which may be relevant to any certain data processing task.
Multi-threaded processing arrangements have been proposed which enable parallel processing to be implemented. Processing data in parallel (e.g. using multiple processing threads) may enable a reduction in the time required to perform a data processing task by distributing the workload associated with the task across processing engines.
However, there remains the problem of how to distribute processing workload across several separate threads/processes. For example, an approach which enables the processing workload to be distributed evenly may be desired, but such a distribution may not be optimal in terms of performance for example. Also, there may exist a conflict between the need to decrease overhead associated with processing workload decomposition and distribution across multiple processing engines, versus the desire to increase potential parallelism for improved load balancing and utilization of the processing engines